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GRE® revised General Test in a Singaporean University 
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International institutions have been increasingly using the GRE® revised General Test to admit students to graduate programs. However, 
little is known about how scores from the GRE revised General Test are used in the admission process outside of the United States and 
their validity in predicting graduate students’ performance (e.g., their graduate school grade point averages [GGPAs]). As the GRE 
revised General Test was launched in August 2011, there is a compelling need to investigate its predictive validity, particularly in an 
international context. A large percentage of examinees who take the GRE revised General Test from outside of the United States are 
citizens of Asian countries. Consequently, we examined how scores from the GRE revised General Test predict a range of graduate 
student performance outcomes at a Singaporean institution that represents the highest caliber of academic excellence in Asian countries. 
We also interviewed key members of the admissions committees to understand how the GRE revised General Test and its individual 
sections are used in the admission process. Our analyses revealed that scores from the GRE revised General Test predicted GGPA and 
program standing. In particular, these scores showed incremental value beyond undergraduate GPA (UGPA) for predicting GGPA. 
Furthermore, among enrolled students, those who submitted scores from the GRE revised General Test in application had significantly 
higher GGPAs than those who did not. These findings largely apply to both doctoral and master’s students. 

Keywords GRE; validity; predict; graduate grade point average; GGPA; Singapore 
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The predictive validity of the GRE® revised General Test 1 has been extensively studied. Researchers have examined the 
GRE’s prediction of first-year grade point average (GPA), graduate GPA (GGPA), and faculty ratings for various disci¬ 
plines, populations, and situations, and differing levels of degrees (Klieger, Cline, Holtzman, Minsky, & Lorenz, 2014; 
Kuncel & Hezlett, 2007; Kuncel, Hezlett, & Ones, 2001; Kuncel, Wee, Serafin, & Hezlett, 2010; Powers, 2004). A revised 
version of the GRE General Test was launched in August 2011. Therefore, it is essential to investigate the use and predictive 
validity of the recently released GRE. Furthermore, the GRE might be very useful to graduate and professional programs 
located outside of the United States in making admissions and other types of decisions (e.g., awarding certain types of 
financial support). Most of the previous investigations focused on the US population without much consideration of the 
GRE’s use in an international context. Given the rapid globalization of higher education, it is important to examine the 
GRE’s prediction of graduate school performance outcomes in foreign countries where the GRE increasingly has been 
used to facilitate admissions decisions. 

In particular, little is known about how GRE scores are used in Asian countries and their validity for predicting stu¬ 
dents’ performance outcomes in Asian graduate programs. Singaporean institutions receive a significant volume of GRE 
scores each year, and most of the scores are sent to a leading institution in Singapore: the National University of Singapore 
(NUS). NUS is ranked in the top 50 among higher education institutions in the world and represents the highest levels 
of academic excellence in Asia (Quacquarelli Symonds, 2012a). For example, NUS was ranked as the second best uni¬ 
versity in Asia in 2012 by Quacquarelli Symonds (2012b), a company that provides information on higher education and 
career choices. Given the possible differences in applicants, student bodies, settings of graduate programs, and criteria for 
graduate student performance outcomes, it is necessary to understand how GRE scores are used in Asian institutions and 
GRE scores’ prediction of graduate students’ performance. Our preliminary research, which included analyzing counts 
of GRE score reports and contacting possible candidate institutions, indicates that Singapore is one of the few regions in 
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Asia likely to provide enough information to make this type of research feasible. In addition, English is the main language 
of instruction at NUS. 

This study addresses three research questions: 

1. What is the predictive validity of GRE scores for predicting students’ performance outcomes in graduate and pro¬ 
fessional programs at NUS? 

2. Is there differential predictive validity between students in master’s versus doctoral programs? 

3. To what extent and how are GRE scores used by Singaporean institutions and/or programs in evaluating candidates? 

Literature Review 

A wealth of research exists that documents the predictive validity of standardized tests for use in admissions to US graduate 
programs. Kuncel and Hezlett (2007) conducted a comprehensive review of standardized tests representing a variety of 
disciplines, such as business, law, medicine, pharmacy, and other domains, and concluded that standardized test scores will 
predict a wide range of performance outcomes, including GPA, degree attainment, licensing examination performance, 
research productivity, and faculty evaluation. Among the standardized tests, the earlier version of the GRE is one of the 
most widely used and researched programs. The scores have been required by more than 90% of the doctoral programs 
and more than 80% of the master’s programs in the United States (Norcross, Hanych, & Terranova, 1996). GRE scores are 
also often used to facilitate decisions about fellowships and other types of financial awards (Rock & Adler, 2014). 

Research to date has used a common set of criteria when investigating predictive validity of the previous version of 
the GRE. These criteria include (a) GPA, both first year and cumulative; (b) comprehensive examination performance; (c) 
faculty rating of students; (d) research productivity (publications, conference presentations, and citation counts); and (e) 
degree attainment and time to degree completion (e.g., Enright & Gitomer, 1989; Kuncel et al., 2001, Kuncel et al., 2010; 
Powers, 2004; Powers & Fowles, 2000). 

McCloy, Campbell, and Cudeck’s (1994) theory of declarative knowledge, procedural knowledge, and motivation has 
been used to explain GRE scores’ prediction of graduate school performance. Declarative knowledge is about knowing 
what to do, procedural knowledge is about knowing how to do it, and motivation is the determination and persistence 
associated with completing a task (Kuncel et al., 2001). As the GRE is a measure of general cognitive skills, it captures the 
essence of declarative and procedural knowledge, and GRE scores reflect individual differences in motivation to pursue 
graduate education. Another theoretical framework that explains the GRE’s prediction of graduate school performance 
outcomes is that the general cognitive ability assessed by the GRE has positive and strong relationships with the ability 
to acquire functional knowledge (i.e., job knowledge) required for successful graduate school performance (Borman, 
Hanson, Oppler, Pulakos, & White, 1993; Schmidt & Hunter, 1993). One would expect that graduate students with greater 
functional knowledge would act more efficiently and effectively as they advance through a program. 

A large-scale meta-analysis with 1,753 independent samples of more than 80,000 students reported that both GRE 
Quantitative Reasoning scores (GRE-Q) and GRE Verbal Reasoning scores (GRE-V) were strong predictors of GGPA 
( p = .32 and .34, respectively), comparable to the undergraduate GPA (UGPA) prediction of GGPA (p = .30; Kuncel et al., 
2001). Note that p stands for the mean correlations adjusted for range restriction and unreliability of the measures of 
graduate school performance. The GRE® Subject Tests were stronger predictors of GGPA (p = .41) than the GRE. The GRE 
Subject Tests also substantially predicted students’ comprehensive exam scores (p = .51). The GRE-Q and GRE Subject 
Tests were also moderate predictors of publication citation count (p = .23 and .24, respectively). 

Klieger et al. (2014) examined the predictive validity of GRE scores (from the prior version of the test) for predicting 
GPA in graduate program areas at 10 Florida public universities in the United States. In addition to its large dataset (21,127 
master’s students and 4,229 doctoral students) covering 28 disciplinary domains as well as its use of several analytical 
approaches, this study uniquely contributed to our understanding of the GRE as the first study to examine the predictive 
validity of GRE Analytical Writing (GRE-AW) scores. In the study, the validity of GRE-AW for predicting GGPA was 
.19 for master’s students and .21 for doctoral students. These validity coefficients were corrected for multivariate range 
restriction but not for measurement error in the predictor or criterion. In general, GRE-AW predicted GGPA as strongly 
as GRE-V and GRE-Q did—and sometimes even more robustly. 

Powers (2004) investigated the validity of the GRE for admissions to colleges of veterinary medicine using data from 
16 institutions. After correcting for range restriction, by considering the GRE scores of both the admitted students and 
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applicants, and unreliability in the criterion (i.e., first-year GPA), he found that the correlation between first-year GPA 
and GRE measures was .30 for GRE-V and .44 for GRE-Q, respectively. GRE scores and UGPA jointly accounted for 
approximately 65% of the variance in first-year GPA. 

Many researchers have examined subgroup differences in their evaluation of the GRE’s predictive validity. Kuncel et al. 
(2001) compared the relationship between the GRE and a number of graduate school performance outcomes across dis¬ 
ciplinary domains, including humanities, social sciences, life sciences, and math and physical sciences. The prediction of 
GGPA by GRE scores was fairly similar across the disciplinary areas, with ps ranging from .27 to .42. A notable finding 
is that the GRE-Q (p = .38) was a stronger predictor for the math and physical science majors than the GRE-V (p = .30), 
whereas the GRE-V (p = .32) was a better predictor for humanities majors than the GRE-Q (p = .27). In predicting faculty 
ratings of student ability, professional work, potential, or overall performance, GRE-V was a substantially large predictor 
for humanities students (p = .72), and GRE-Q was a large predictor for the math and physical science students (p = .63). 

In 2005, Burton and Wang investigated the validity of GRE-V and GRE-Q for predicting several graduate school cri¬ 
teria. For forecasting mastery of discipline, professional productivity, communication skills, and cumulative GPA across 
several disciplines, the authors calculated multiple correlations for GRE-V and GRE-Q after corrections were made for 
multivariate range restriction. These corrected R values ranged from .40 to .52 (.30-.37 uncorrected; N = 319- 1,303 for 
10-19 graduate departments). For the same criteria, they also looked at similarly corrected multiple correlations specifi¬ 
cally for the graduate disciplines of biology (.51 - .56; n = 145 across five departments), chemistry (.17- .52; n = 134 across 
two departments), education (.32-.66; n = 699 across three departments), English (.44-.62; n = 170 across five depart¬ 
ments), and psychology (.26- .51; n = 155 across four departments). 

Kuncel et al. (2010) conducted a meta-analysis of nearly 100 studies and 10,000 students in examining the differential 
predictive validity of the GRE for master’s and doctoral programs. The performance outcome measures included first- 
year GPA, GGPA, and faculty ratings. The authors reported that GRE scores predict the three types of outcomes well for 
both master’s and doctoral students. A difference in GRE-V’s prediction of GGPA was noted between these two groups of 
students, with a stronger prediction observed among master’s students. The authors suggested that the difference may be 
a result of the smaller range of grades for doctoral students (S.D. = .21) than master’s students (S.D. = .40). 

There has been some prior validation of the GRE revised General Test in a non-US context, but it has been limited thus 
far to students seeking a master’s degree in a European context. Schwager, Hiilsheger, and Lang (2014) and Schwager, Hiil- 
sheger, Lang, and Bridgeman (2014) examined the validity of the GRE for predicting average GPA and degree attainment 
for 282 multinational students enrolled at a Dutch university. For the entire sample ( N = 236), each GRE section individu¬ 
ally and statistically significantly predicted GGPA (r = .17 — .31, p < .01), and GRE-AW statistically significantly predicted 
master’s thesis grade (r= .15, p < .05, N = 234). In a regression model accounting for disciplinary area, the three GRE 
sections collectively predicted GGPA over and above UGPA, English proficiency, and socioeconomic status (AR 2 = .12, 
p < .001, N = 234). Also, the GRE sections predicted master’s thesis grade over and above UGPA, English proficiency, and 
socioeconomic status (AR 2 = .05, p < .01, N = 232). For predicting degree attainment and time to complete the master’s 
degree, results for subtests individually and combined were not statistically significant (p > .05, N = 233-279). 

Although the NUS is a government-funded public university in Singapore, the Singaporean Ministry of Education 
(MOE) gives public universities great autonomy in their internal governance, including over decisions of admission 
requirements (Chan, 2012). One of the significant goals of the MOE is to develop world-class higher education and 
make Singapore the “Boston of Asia” (Duhamel, 2004). Following the American “gold standard,” the graduate admis¬ 
sions requirements at NUS are very similar to those requirements at US institutions. For example, for admissions to most 
research programs at the graduate level, a list of requirements includes academic degree and record, recommendation let¬ 
ters, standardized test scores, and demonstration of relevant work-study experience (NUS, n.d.). Proficiency of English 
as indicated by scores on the TOEFL® test or other similar standardized tests is also required because the medium of 
instruction is English. 


Methodology and Design 

The GRE revised General Test 

We gathered data on students’ GRE scores, including section scores for the three GRE sections (GRE-V, GRE-Q, and 
GRE-AW). The GRE revised General Test was launched in August 2011 and is a standardized test used by graduate and 
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professional schools to make admissions decisions (Educational Testing Service [ETS], 2012). Some graduate programs 
may use the exam to determine which admitted students receive fellowship awards. The test, which is timed, typically takes 
approximately 4 hours to complete. The GRE consists of three major types of items designed to assess different skills and 
abilities: verbal reasoning, quantitative reasoning, and analytical writing. Both verbal reasoning and quantitative reasoning 
use multiple-choice items, among others, whereas analytical writing uses constructed-response items. 

Verbal reasoning and quantitative reasoning each have two sections of items, and each section contains 20 questions. 
Verbal reasoning measures the extent to which the examinee comprehends what the examinee reads and how he or she 
uses reasoning skills. Quantitative reasoning, with an increased focus on data interpretation, assesses the examinee’s ability 
to interpret, apply, and solve mathematical problems. The GRE adopts a multistage design in that for the second set of the 
verbal reasoning and quantitative reasoning sections, respectively, an examinee will be routed to one of three panels of 
items of low, medium, or high difficulty, depending on his or her performance on the first section. The multistage design 
is expected to improve the accuracy of examinees’ ability estimate. Analytical writing, which consists of two constructed- 
response tasks, requires written responses that analyze an issue or argument to measure the examinee’s ability to articulate 
and support his or her ideas in English. 

The exam comes in both computer-based and paper-based formats, and the majority of examinees take the computer- 
based version. The computer-based GRE permits the examinee to preview and review items within a section, tag questions, 
and edit responses within a section. The quantitative reasoning section now provides an on-screen calculator for making 
computations to reduce the test’s focus on computational ability. In addition, the GRE includes new answer formats such 
as numeric entry. 

Sample 

Among the 17,206 NUS applicants, 7,241 (42%) were females and 9,965 (58%) were males. There were 4,450 applicants 
for the 2011-2012 academic year (26% of total), 5,778 for 2012-2013 (34% of total), and 6,978 for 2013-2014 (41% of 
total). Of the 17,206 applicants, 3,371 (20%) were admitted and enrolled. No information was available about the number 
of admitted applicants who enrolled elsewhere or which applicants were applying for doctoral programs versus master’s 
programs. Of the matriculants, 924 (27%) enrolled in the 2011-2012 academic year, 1,192 (35%) in the 2012-2013 aca¬ 
demic year, and 1,255 (37%) in the 2013-2014 academic year. On the basis of the data available, we could identify degree 
level (master’s or doctoral) for 86% of the enrolled students; we identified 1,923 enrollees as seeking master’s degrees and 
991 enrollees as seeking doctoral degrees. Among the enrolled students, 1,537 (46%) were females and 1,834 (54%) were 
males. Enrolled students comprised 2,965 (88%) full-time and 406 (12%) part-time students. 

Indicators of Student Background and Graduate Student Performance Outcomes 

In addition to GRE scores, we gathered data on student background and multiple student performance outcomes in gradu¬ 
ate and professional programs: (a) background variables (gender, ethnicity, citizenship, language, class status, major field 
of study); (b) first-semester and first-year GGPAs; and (c) program standing. All data were obtained from the official 
academic records of students from the registrar’s office. 

Survey and Interview 

We invited NUS faculty and administrators to take a 10-minute online survey to ascertain their perceptions and use of the 
GRE and the larger graduate admissions process at NUS. The online survey appears in Appendix A. We received responses 
from 11 individuals, 10 of whom were interviewed or participated in focus groups discussed in the following paragraphs. 

Similar to the approach used in a previous ETS investigation of GRE score use by US graduate schools (see Walpole, 
Burton, Kanyi, & Jackenthal, 2002), we interviewed or conducted focus groups with 21 NUS graduate faculty and admin¬ 
istrators who participated in or were very familiar with admissions and funding of graduate students. Our NUS contact 
arranged for these meetings after considering our need for a representative sample of graduate faculty. The script that 
was used in interviews and focus groups appears in Appendix B. Some of the participants had already completed surveys, 
so the interviews or focus groups served as follow-up to collect additional information. Some of the participants were 
new, and we elicited from them survey information as well as supplementary information. After the interviews and focus 
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groups were completed, there were no nonresponses. To arrive at qualitative findings, content analysis was conducted to 
identify crosscutting themes across the survey, interview, and focus groups. 

Statistical Analysis 

Descriptive statistics such as means and standard deviations were provided for the applicants and enrollees, as were the 
effect sizes of the differences between these two groups. Effect sizes indicated by Cohen’s (1988) d were also reported for 
all t-tests. 

Correlation and regression analyses were conducted to determine the direct and incremental validity of using GRE 
scores to predict multiple performance outcomes variables. In terms of direct predictive validity, we calculated zero-order 
correlation coefficients (rs in this report) between each predictor (e.g., GRE-V) and performance outcome criteria (e.g., 
first-year GGPA). We also adjusted these validity coefficients for range restriction of the samples. Range restriction refers to 
the fact that when only part of a range of scores for a variable is considered, the correlation between this variable and other 
variables tends to be artificially depressed (Pearson, as cited in Powers, 2004). When evaluating the relationship between 
GRE scores and graduate school performance outcomes, only those enrolled in the programs normally are included in 
the analysis, because we lack criterion scores for those who do not enroll. Ideally, data for all program applicants should 
be included, because we actually want to assess predictive validity for the applicant pool. Research has shown that range 
restriction could have a substantial impact on the correlation coefficients (Kuncel et al., 2001; Linn & Dunbar, 1982). We 
employed univariate approaches of adjusting for direct range variation recommended by Hunter and Schmidt (2004). The 
formula appears in Appendix C. 

In terms of incremental validity, we employed multivariate predictive validity metrics such as multiple correlations 
(Rs) and incremental coefficients of determination (AR 2 s) for the prediction of those same criteria. We were specifically 
interested in knowing GRE scores’ incremental predictive validity over and above UGPA. To calculate incremental validity, 
with UGPA, we conducted regression analyses that included the three GRE scores together and individually. AR 2 s indicate 
incremental prediction, the amount of variance in GGPA explained by GRE scores over and above what is explained 
by UGPA. 

Because the applicants were from hundreds of international undergraduate programs with different GPA scales (e.g., 
0-4,0-5,0-10,0 -100), it was necessary to standardize UGPAs for the regression analyses described earlier. However, for 
many applicants, information on which scales their UGPAs were based were missing. As a result, only a subset consisting 
of 135 applicants whose UGPAs could be confidently converted to a 0 -4 scale was included in the incremental prediction 
analyses. 2 

The reliability of the criterion variables (Cronbach’s alpha based on three semesters) was .71. As the reliability is less than 
1, it attenuates the validity coefficient for the predictor, even though the predictor is not responsible for that unreliability 
(Hunter & Schmidt, 2004; Spearman, 1904). Nevertheless, we chose not to adjust coefficients for the unreliability of the 
criteria for the reason that the GRE subtests are not responsible for the attenuation of correlation coefficients caused by 
unreliability of criteria and that unreliability is part of operational reality. Using regression methods, we estimated the 
validity of the GRE subtests when used together as well as the incremental validity over other predictors (e.g., UGPA) of 
GRE subtests when used together and separately. 

As degree type (i.e., master’s or doctoral) may interact with the GRE prediction of performance outcomes in graduate 
programs (Kuncel et al., 2010), subgroup analyses were conducted to study differences with regard to degree. 

Results 

Test Scores of the Enrolled Students and Applicants 

Table 1 shows the descriptive statistics of the GRE information for both applicants and enrolled students. On average, 
enrolled students performed significantly better than applicants on the verbal reasoning and analytical writing sections 
(p < .001). The three histograms in Figure 1 show that the results are consistent with this conclusion for the verbal rea¬ 
soning section; that the quantitative reasoning scores are skewed toward the high end; and that a somewhat more even 
(less peaked) spread of analytical writing scores for enrolled students compared to applicants are conveyed, but given the 
particularly discrete nature of scoring on the 0-6 analytical writing scale, a clearly trimodal score distribution is observ¬ 
able for both enrolled students and applicants. Based on Cohen’s (1988) guidelines, both sections also showed small effect 
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Table 1 Comparisons Between GRE Scores for Applicants and Enrolled Students 




Applicants 



Enrolled 


t 

d 

N 

M 

S.D. 

N 

M 

S.D. 

Overall 









GRE-V 

2,985 

150.70 

6.47 

485 

152.29 

6.28 

5.03"* 

0.25 

GRE-Q 

2,985 

162.47 

12.89 

485 

163.20 

5.42 

1.23 

0.06 

GRE-AW 

2,917 

3.38 

0.58 

468 

3.54 

0.60 

5.37*" 

0.27 

Master’s 









GRE-V 

a 



224 

152.43 

6.30 

3.87"* 

0.27 

GRE-Q 




224 

162.73 

5.33 

0.30 

0.02 

GRE-AW 




213 

3.55 

0.61 

4.17*** 

0.30 

Doctoral 









GRE-V 




261 

152.16 

6.27 

3.51*** 

0.23 

GRE-Q 




261 

163.61 

5.48 

1.41 

0.09 

GRE-AW 




255 

3.52 

0.59 

3.74”* 

0.24 


Note. GRE-V = verbal reasoning scores; GRE-Q = quantitative reasoning scores; GRE-AW = analytical writing scores. 
“Data were not available to differentiate between doctoral and master’s applicants. 

"><. 001 . 


Revised GRE Verbal scores for Enrolled vs Applied 
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Revised GRE Analytical Writing scores for Enrolled vs Applied 



Figure 1 Comparisons between GRE Verbal Reasoning (GRE-V), GRE Quantitative Reasoning (GRE-Q), and GRE Analytical Writing 
(GRE-AW) scores for applicants and enrolled overall, masters, and doctoral students. 


sizes in terms of differences. Similar results are also reported for enrolled master’s and doctoral students in comparison to 
all applicants (see Table 1 and Figure 1). Findings are similar in that both enrolled master’s and enrolled doctoral students 
performed significantly better than applicants on the verbal reasoning and analytical writing sections (p < .001), and the 
effect sizes were small. Applicants and enrolled master’s and doctoral students did not show any statistically significant 
performance difference on the quantitative reasoning section, and the magnitude of the difference was also negligible. 

Although the three histograms in Figure 1 visually indicate some score distributional differences between our samples 
of master’s and doctoral students, the results in Table 1 quantitatively show that the mean GRE scores of enrolled master’s 
students differ little from the mean GRE scores of enrolled doctoral students. This finding varies considerably from what 
one usually observes for graduate programs in the United States, where mean GRE scores for students enrolled in doc¬ 
toral programs — generally more selective than master’s programs — are nontrivially higher than the mean GRE scores for 
students enrolled in master’s programs. It is possible that the observed difference in the United States versus Singapore is 
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Table 2 Correlations Between GRE Scores and Graduate Grade Point Averages (GGPAs) Unadjusted and Adjusted for Range 
Restriction 



Unadjusted 



Adjusted 



First-semester GGPA 

First-year GGPA 

First-semester GGPA 

First-year GGPA 

Overall 

GRE-V 

.21” 

.20** 

.21 


.20 

GRE-Q 

.16” 

.19” 

.29 


.32 

GRE-AW 

.15” 

.14” 

.14 


.14 

Master’s 

GRE-V 

.18* 

.18* 

.19 


.18 

GRE-Q 

.31” 

.32** 

.48 


.49 

GRE-AW 

.10 

.12 

.10 


.12 

Doctoral 

GRE-V 

.26** 

.25** 

.27 


.26 

GRE-Q 

.01 

.02 

.01 


.03 

GRE-AW 

.21” 

.19” 

.21 


.19 


Note. N = 381-404, overall; 173-184, masters; 203-215, doctorate. GRE-V = verbal reasoning scores; GRE-Q = quantitative reasoning 
scores; GRE-AW = analytical writing scores. 

*p < .05. **p<.01. 


related to cross-national demographic differences in who applies, and thus who is admitted, to doctoral versus master’s 
programs. Future research is needed to unveil the reasons that could explain the similar mean scores between the enrolled 
degree levels at this Singaporean institution. 


Prediction of Graduate Grade Point Average 

The GRE scores are significantly correlated with a number of key graduate school performance outcomes. Specifically, 
all three GRE sections (verbal reasoning, quantitative reasoning, and analytical writing) are significantly correlated with 
first-semester GPA and first-year GPA (see Table 2). After adjusting for range restriction, the correlations increased sub¬ 
stantially for GRE-Q and, with the exception of a small decrease for GRE-AW predicting first-semester GPA, remained 
unchanged for GRE-V and GRE-AW (see Table 2). 

Results by graduate degree level indicate that across GPA criteria, for master’s students, GRE-Q was predictive, but 
the predictive validity of GRE-AW was not statistically significant (Table 2). For doctoral level students, GRE-V and 
GRE-AW—but not GRE-Q—were statistically significant predictors across GPA criteria (Table 2). 


Prediction of Program Standing 

GRE examinees in good program standing (who never received an academic warning, academic probation, or dismissal) 
scored higher on average than those GRE examinees who were not (see Table 3). But those mean differences in GRE scores 
were not statistically significant. Based on Cohen’s (1988) guidelines, the effect sizes are small (see Table 3). 

Results by degree show that these findings hold up for master’s students specifically, except for GRE-AW, where mean 
GRE scores were lower, but to a nonstatistically significant extent for those who were in good standing (see Table 3). Based 
on Cohen’s (1988) standards for judging the magnitude of differences, the effects for the difference in GRE scores between 
those who had a favorable graduate program standing and those who did not were negligible for GRE-V and GRE-AW 
but large for GRE-Q (see Table 3). Note that sample sizes were very small for those not in good standing (n = 11). 

Except for GRE-Q, the overall findings also hold for doctoral students (where mean scores were lower to a nonstatis¬ 
tically significant extent for those who were not in good standing; Table 3). The effects for the difference in GRE scores 
between those who had a favorable graduate program standing and those who did not were small to medium in size for 
GRE-V and GRE-AW but negligible for GRE-Q (Table 3). The latter finding for GRE-Q might be attributable to the small 
number (n = 20) of doctoral students on academic warning. 
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Table 3 Differences in Mean GRE Scores Based on Program Standing 




Good standing 



Not good standing 





N 

M 

S.D. 

N 

M 

S.D. 

t 

d 

Overall 

GRE-V 

461 

152.32 

6.33 

31 

150.55 

6.03 

1.52 

0.28 

GRE-Q 

461 

163.17 

5.52 

31 

162.10 

5.18 

1.05 

0.19 

GRE-AW 

444 

3.54 

0.60 

31 

3.40 

0.52 

1.27 

0.24 

Masters 

GRE-V 

213 

152.48 

6.34 

11 

151.55 

5.52 

0.48 

0.15 

GRE-Q 

213 

162.95 

5.29 

11 

158.45 

4.25 

2.77" 

0.86 

GRE-AW 

202 

3.55 

0.61 

11 

3.59 

0.54 

-0.21 

-0.06 

Doctoral 

GRE-V 

241 

152.34 

6.24 

20 

150.00 

6.37 

1.61 

0.37 

GRE-Q 

241 

163.56 

5.55 

20 

164.10 

4.58 

-0.42 

-0.10 

GRE-AW 

235 

3.54 

0.60 

20 

3.30 

0.50 

1.76 

0.41 

Note. GRE-V 

= verbal reasoning scores; GRE-Q 

= quantitative reasoning scores; GRE-AW = 

analytical writing 

scores. 



*><. 01 . 


Table 4 Incremental Validity of Optimally Weighted GRE Sections Together Over Undergraduate Grade Point Average (UGPA) 


R 2 


GGPA first semester GGPA first year 


Overall 

UGPA 

.03 

.02 

UGPA + GRE-V + GRE-Q + GRE-AW 

.09 

.10 

A R 2 

.07 

.07 

Masters 

UGPA 

.00 

.00 

UGPA + GRE-V + GRE-Q + GRE-AW 

.15 

.15 

AR 2 

.15 

.15 

Doctoral 

UGPA 

.04 

.04 

UGPA + GRE-V + GRE-Q + GRE-AW 

.06 

.07 

AR 2 

.03 

.03 


Note. The sample size here for UGPA was 135. As noted in the text, UGPA from only these 135 students could be confidently converted 
to a 0 - 4 scale. GGPA = graduate grade point average; GRE-V = verbal reasoning scores; GRE-Q = quantitative reasoning scores; GRE- 
AW = analytical writing scores. 


Incremental Prediction Over Undergraduate Grade Point Average 

GRE scores were added to the regression including UGPA to investigate the incremental predictive validity of GRE scores. 
Results (see Table 4) provide evidence of GRE scores’ incremental predictive validity with various GGPA variables as the 
criteria. Note that AR 2 values in Table 4 have been rounded to two digits after subtraction involving more than two digits. 
When GRE scores were added to the regression, the change in R 2 ranged from .07 to .08 for the overall sample. For master’s 
students, the change in R 2 was even more substantial (.15), which suggests that GRE scores were a much stronger predictor 
of GGPA than was UGPA (see Table 4). The incremental prediction of GRE scores for doctoral students in terms of the 
R 2 ranged from .03 to .06 (see Table 4). 

Tables 5-7 show the results when separate GRE scores were included in the regression. GRE-V and GRE-Q scores pro¬ 
vided incremental prediction of various GGPAs for the overall sample. In addition, GRE-Q scores provided stronger incre¬ 
mental prediction for master’s students (see Table 6), whereas GRE-AW scores showed stronger incremental prediction 
for doctoral students (see Table 7). GRE-Q’s incremental prediction was stronger for master’s students in that the change 
in R 2 ranged from .08 to .12 (see Table 6). 
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Table 5 Incremental Validity of Optimally Weighted GRE Verbal Reasoning (GRE-V) Over Undergraduate Grade Point Average 
(UGPA) 




R 2 



GGPA first semester 


GGPA first year 

Overall 

UGPA 

.03 


.02 

UGPA + GRE-V 

.04 


.04 

A R 2 

.01 


.02 

Master’s 

UGPA 

.00 


.00 

UGPA + GRE-V 

.01 


.03 

A R 2 

.01 


.03 

Doctoral 

UGPA 

.04 


.04 

UGPA + GRE-V 

.05 


.05 

AR 2 

.02 


.02 


Note. GGPA = graduate grade point average. 


Table 6 Incremental Validity of Optimally Weighted GRE Quantitative Reasoning (GRE-Q) Over Undergraduate Grade Point Average 
(UGPA) 


R 2 



GGPA first semester 

GGPA first year 

Overall 

UGPA 

.03 

.02 

UGPA + GRE-Q 

.08 

.08 

A R 2 

.05 

.05 

Masters 

UGPA 

.00 

.00 

UGPA + GRE-Q 

.12 

.10 

AR 2 

.12 

.10 

Doctoral 

UGPA 

.04 

.04 

UGPA + GRE-Q 

.05 

.05 

AR 2 

.02 

.02 


Note. GGPA = graduate grade point average. 


Difference in Graduate Grade Point Average With or Without GRE Scores 

Those students who took the GRE test obtained higher GGPAs (first semester and first year), on average, than those 
students who did not (see Table 8 and Figure 2). However, these differences were statistically significant for first-semester 
and first-year GGPAs only (p < .05; see Table 8). The effect sizes for the difference in GRE scores between those who took 
and did not take the GRE examination were medium across GRE sections (see Table 8 and Cohen, 1988). 

The results also hold up for master’s students with statistical significance (see Table 8). The effects for the difference in 
GRE scores between those who took the GRE test and those who did not were small across GRE sections (see Cohen, 1988). 

For doctoral students, GGPAs were higher for those who took the GRE, but none of the mean score differences between 
those who took the test and those who did not was statistically significant (see Table 8). The effect sizes for the difference 
in GRE scores between those who took the GRE test and those who did not were negligible across GRE sections (see 
Table 8). For both first-semester and first-year GGPA distributions (as shown in Figure 2), the histograms are bimodal for 
doctoral students (but unimodal for master’s students) whether or not the doctoral students took the GRE test; however, 
the bimodality is more pronounced for doctoral students who took the test (also shown in Figure 2). It is possible that there 
is more than one enrolled doctoral student subpopulation, but owing to a lack of data necessary to substantively explore 
further the nature of the bimodality, we recommend that additional studies be conducted to investigate this observation. 
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Table 7 Incremental Validity of Optimally Weighted GRE Analytical Writing (GRE-AW) Over Undergraduate Grade Point Average 
(UGPA) 


R 2 



GGPA first semester 

GGPA first year 

Overall 

UGPA 

.03 

.02 

UGPA + GRE-AW 

.03 

.03 

A R 2 

.00 

.01 

Masters 

UGPA 

.00 

.00 

UGPA + GRE-AW 

.00 

.01 

A R 2 

.00 

.01 

Doctoral 

UGPA 

.04 

.04 

UGPA + GRE-AW 

.04 

.05 

A R 2 

.00 

.01 


Note. GGPA = graduate grade point average. 


Table 8 Differences in Mean Graduate Grade Point Averages (GGPAs) Between Those Who Took and Did Not Take the GRE 




No GRE 



GRE 





N 

M 

S.D. 

N 

M 

S.D. 

t 

d 

Overall 

First-semester GPA 

1,775 

3.76 

0.68 

398 

4.08 

0.62 

8.66“ 

0.48 

First-year GPA 

1,827 

3.77 

0.63 

401 

4.10 

0.59 

9.51“ 

0.48 

Masters 

First-semester GPA 

1,154 

3.76 

0.63 

184 

3.94 

0.58 

3.69“ 

0.29 

First-year GPA 

1,160 

3.78 

0.59 

184 

3.93 

0.56 

3.41“ 

0.27 

Doctoral 

First-semester GPA 

260 

4.16 

0.62 

209 

4.20 

0.63 

0.84 

0.08 

First-year GPA 

267 

4.21 

0.53 

212 

4.24 

0.58 

0.65 

0.06 


'"p < .001 


Main Findings From the Survey, Interviews, and Focus Groups 

Table 9 presents information regarding the department, admissions role, type of degree program, citizenship, and gen¬ 
der of the focus group participants. Participants were well representative of NUS graduate faculty in general and of the 
subject areas one would encounter at the graduate level at most research universities. A majority of the respondents in 
the study were from the science, technology, engineering, and mathematics (STEM) disciplines, including physics, chem¬ 
istry, biology, mathematics, and engineering. A substantial minority were from arts and social science programs, which 
included psychology, English literature, and social work. Representatives from design and environment taught and con¬ 
ducted research in architecture, real estate, and building. Respondents included representatives from the business school 
and law school, which unlike US business and law schools, do not have a long tradition of using specific standardized 
assessments for admissions. 


To What Extent Is the GRE revised General Test Required or Recommended? 

Eleven respondents (52% of the 21 surveyed individuals) indicated that the GRE was required for their program but can 
be waived under certain circumstances. Five of the respondents (24%) indicated that the test is required or recommended 
for some degree students. None of the respondents indicated that the GRE is required of all applicants. 

Five individuals (24%) commented that the GRE is required for foreign (non-Singaporean) students or students who 
have not graduated from the three top Singaporean universities (NUS, Nanyang Technological University, and Singapore 
Management University). The finding is consistent with the interview results. In fact, one interviewee said that it is the 
policy of the university that any student from a foreign university other than a US university is required to submit GRE 
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1st Semester Graduate GPA for Non-GRE and Revised GRE Students 



□ Revised GRE Ovwal 
n Non-GRE 0.»al 


j 'Revised GRE 

f 1 


Revert GRE Doclcnl 
j Non-GRE UHeti 
Non-GRE Doeloral 



Figure 2 Comparisons of first-semester and first-year graduate grade point average (GGPA) distributions between takers of the GRE 
and those who took no GRE test. 


scores. Others said that the GRE is required for non-Singaporean students, although certain groups of foreigners may be 
exempt from this requirement. For instance, one respondent mentioned that students graduating from top-ranked foreign 
universities, especially those in the United States and United Kingdom, may not be required to take the GRE. 

In the interviews, some respondents explained the reason for requiring the GRE from foreign applicants. Foreign stu¬ 
dents (or, presumably, students who have not graduated from the three top Singaporean universities) have attended a 
variety of institutions, the quality of which is unknown to admissions committees. In these cases, the GRE provides a 
standard measure against which foreign students’ academic ability can be compared as well as a context within which 


GRE Board Research Report No. 16-01 and ETS Research Report Series No. RR-16-05. © 2016 Educational Testing Service 


11 














0. L. Liu ef al. 


Predictive Validity of Scores From the GRE® revised General Test in a Singaporean University 


Table 9 Characteristics of Participants (21 Respondents) 



jV a 

% 

Department 

21 


Science, technology, engineering, and mathematics 

12 

57.1 

Art and social science 

4 

19.0 

Design and environment 

3 

14.3 

Business 

1 

4.8 

Law 

1 

4.8 

Graduate committee 

23 


Served as member 

11 

47.8 

Served as chair 

9 

39.1 

Other 

2 

8.7 

Have not served 

1 

4.3 

Graduate degree program 

26 h 


Research doctorate 

17 

65.4 

Research masters 

4 

15.4 

Professional master’s 0 

4 

15.4 

Professional doctorate 

1 

3.8 

Citizenship 

21 


Undisclosed 

11 

52.4 

Republic of Singapore 

6 

28.6 

Republic of Korea 

1 

4.8 

Malaysia 

1 

4.8 

United States 

1 

4.8 

United Kingdom 

1 

4.8 

Gender 

21 


Male 

15 

71.4 

Female 

6 

28.6 


M 

SD 

Age (years) 

46.3 

9.45 

Years as chair of graduate committee 

3.1 

2.03 

Years on graduate committee 

6.45 

4.23 


a ]V = number of responses. As the 21 participants could provide multiple answers, the value of N is not necessarily 21. The N of each sub¬ 
section of the table is reported in italics. b A respondent may be affiliated with more than one degree program. c Of the four respondents, 
one represented the business school and one represented the law school. 


their GPAs and other academic performance outcomes can be interpreted. A faculty member in the social work program 
remarked: 

Well, in our department, the main reason why having a GRE score is a requirement is very straightforward. I mean, 
applicants who are not graduates from NUS, NTU, or SMU—if that’s the case, we can’t really tell how rigorous their 
education training was for them, so we can’t really tell how good the applicants are in terms of their academic 
abilities, despite looking at their GPA or CAP, whatever. So that’s why we put the GRE score as a requirement, and 
then it’s just a screening tool. So for us, we just screen those who do not meet the minimum GRE score. 

A faculty member from the information technology program stated: 

But what I’m sensing, though, is that here it’s very much being perceived as if you really did graduate from a 
Singapore quality university like NUS, NTU, SMU, then there’s a perception we know about you already. So if you’ve 
got a first-class honors or a second-class honors, we already know what that means, so there’s no need for any further 
validation. But if you come from somewhere else and we don’t really know what to expect of the quality of your 
education, the GRE is sort of being perceived as a way to provide a backup as it were, to provide a second validation. 

In a focus group for science disciplines, one respondent observed, “Because they all come from different institutions, 
it’s difficult to rate them, so the GRE becomes a common denominator by which we do our rating and the first cut. So it 
is a necessary, but not sufficient, requirement.” 
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The responses indicate that the GRE is perceived as useful and is used when it can alleviate ambiguity in applicants’ 
qualifications and provide a common yardstick to compare applicants. The business and law schools do not require or 
recommend to their applicants that they take the GRE. 

How Important Is the Use of GRE in Admissions Decisions? 

On the basis of survey responses, the GRE was perceived as useful primarily for identifying students who could handle 
graduate school course work. Individual departments have much flexibility regarding the use of the GRE, as this exchange 
indicates: 

interviewer: So is there diversity in terms of how the GRE is looked at and used across your different departments? 

respondent: 1 think that is a standard rule, you know, but each department can be flexible in the way in which they 

administer and use it. 

Nine out of 12 individuals (75%) from STEM programs who responded to survey questions about the importance of 
the GRE sections indicated that they were at least moderately important in decision making. A majority of the six (83%; 
n = 5) respondents from humanities programs and two of the three individuals (66%) from social science programs said 
the same for all three GRE sections. In the survey, the business school respondents indicated that all three GRE sections 
are extremely important (the highest importance rating), even though the business school did not require the test but will 
consider GRE scores if submitted. The interview data indicate that, across graduate programs that do use the GRE, scores 
on the test are not given quite as much weight in admissions decisions as other indicators of academic achievement (a 
student’s undergraduate or previous graduate school grades, academic record, research training, professional experience, 
and performance in face-to-face or Skype-based interviews). When asked directly how important the GRE scores are in 
relation to other things, most said that the scores were not usually considered the most important factor. Many respondents 
talked about how they viewed the one-on-one interview with the applicant as an important way to assess the student’s 
English skills. The nature of the department or program determined the indicators given the most weight in admissions. 
In more applied programs, previous professional experience or the department’s own entrance exam was important; in 
research-oriented departments, previous research experience and publications were extremely important. Match with 
faculty research interests was also mentioned. 

How Is the GRE Used? Do the Departments Use a Cutoff Score? 

Surveys indicated that the GRE is used primarily for making admissions decisions rather than for financial award decisions 
or for academic advising (although there is some use for making determinations about fellowships and assistantships). 
Nine out of 13 survey respondents (69% of the respondents to the question) indicated that the GRE is used in the decision¬ 
making process typically when scores are high enough to compensate for other weak credentials. Survey responses indi¬ 
cated across disciplines (including STEM fields) that when the GRE is used, it is used in a holistic process — rather than one 
based on a specific formula—of combining admissions information. The most commonly expressed policy when an appli¬ 
cant sends several sets of GRE scores is to use the most recent ones. Many respondents reported that their departments 
use GRE scores primarily with a cutoff; how the cutoff scores function in admissions decisions varies across departments 
and programs. In some cases, they serve to screen out students who score below a certain point, whereas in others, the 
role of the cutoff and the circumstances when it is applied depend on the student and on other circumstances. Because 
in many cases, only certain groups of students are required to take the GRE (e.g., foreign applicants) and the cutoffs may 
apply only to these groups. In some departments, both the requirement to take the GRE and the minimum acceptable 
scores are often modified if a student’s other academic credentials that the department considers important are strong. 

Some respondents reported that students must meet a minimum score to be considered for admission. For example, 
one interviewee from a social science program whose department requires all foreign applicants to submit GRE scores 
said that the GRE “is a screening tool. We just screen those who do not meet the minimum score.” Another respondent 
reported that the GRE score must “meet our minimum requirement before we evaluate the application.” One interviewee 
(a faculty member from a STEM program) said that GRE test scores are used as a cutoff only with foreign students who 
come from reputable—but not the top — universities where instruction either might not have been conducted in English 
or where English was not a primary language of the countries in which the universities were located. The faculty member 
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representing STEM fields indicated that the GRE score minimum is 320, with a minimum analytic writing score of 4. 
Another respondent stated, “If anyone gets below 300 for the verbal and quantitative, then we probably wouldn’t meet 
that person.” 

The use of cutoff scores among the represented NUS programs was not unlimited in scope. Although GRE scores 
sometimes establish a floor in admissions, high GRE scores do not guarantee admission. One focus group participant 
stated, “Well, I guess my feeling is that, if somebody got a very high GRE, that will not really add weight to his application. 
But if somebody has a very low GRE, you could say, well, that’s really not too good.” A fellow focus group participant 
affirmed, “I agree with him.” Other interviewees reported that their departments do not have “an absolute cutoff’ but 
rather use the GRE as a useful reference to look at the range of applicants or to help rank applicants relative to each other. 

How Is the Analytical Writing Section Perceived and Used? 

Powers and Fowles (2000) indicated that, at least in a US context, there is a strong need to better understand graduate 
faculty’s and administrators’ perceptions and use of a GRE writing assessment, which had not existed in the GRE General 
Test before 1999. (The verbal reasoning and quantitative reasoning sections, though they have evolved over time, have 
existed for a substantially longer period.) At NUS, there is at least some use of the analytical writing section in admissions, 
but respondents voiced various concerns and used the section in various ways. Some individuals expressed concerns over 
the analytical writing section’s ability to indicate English language analytical writing skills needed for successful graduate 
student performance outcomes at NUS. Other individuals wondered why the change was made from the original analytical 
section to the current writing section. As indicated earlier, one faculty member (out of 12) representing STEM fields 
indicated that his program used a GRE minimum analytical writing score of 4. One interviewee representing design and 
environment programs said that the interviewee looks at the analytical writing section “most.” “For verbal we can always 
call them and [do a] face-to-face interview.” Three or four respondents indicated that, to the extent the GRE is used for 
admissions decisions, the analytical writing section was not especially relevant or irrelevant. One individual in this group 
indicated, “We do not look at one section more than the other. We are not even very particular about the total score.” 

Conclusions and Discussion 

Conclusions 

From a series of analyses examining GRE scores’ prediction of graduate performance, we draw the following conclusions: 
(a) GRE predicted GGPA and program standing in that enrolled students with higher GRE scores tend to have higher 
GGPAs (i.e., first-semester and first-year GPAs) and better program standing; (b) GRE predicted GGPA over and above 
the prediction of GGPA by UGPA alone, which suggests that GRE scores make unique contributions to the prediction of 
GGPAs; and (c) among enrolled students, those who submitted GRE scores had significantly higher GGPAs than those 
who did not. 

The preceding conclusions largely apply to all enrolled students as well as to both doctoral and master’s students. The 
most prominent difference revealed from the by-degree (i.e., master’s vs. doctoral degree) analyses are that GRE-Q showed 
a much larger incremental predictive validity of GGPA over UGPA for master’s students than for doctoral students. 

Through survey and focus group interviews, admissions committee members and faculty reported that GRE scores are 
used in a variety of ways in assisting admissions decisions, with some programs requiring GRE scores whereas others do 
not. When GRE scores are required, in most cases, they are used in a holistic way to evaluate an applicant rather than as 
part of a statistical formula for decision making. Representatives from programs that do not require GRE scores indicated 
that they still evaluate the GRE as a standardized way to evaluate applicants when such scores are submitted. 

Limitation 

This study successfully demonstrated the relationship between GRE scores and the performance outcomes of GGPA and 
program standing. One potential limitation of this study is that it only provided information on how the GRE predicts 
graduate performance for a Singaporean institution. Results may not be generalizable to institutions in other Asian coun¬ 
tries given the heterogeneous nature of Asian countries regarding their social, cultural, and educational systems. Another 
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reason responsible for the limited generalizability of findings is that most institutions in other Asian countries (e.g., China, 
Japan, and Korea) do not require GRE scores for graduate program admissions. 

Implications 

Although for now the study findings may have limited generalizability, they may have a long-term impact on how the 
GRE can and should be used in institutions in other countries. As more Asian institutions are moving toward English 
instruction (e.g., top business schools in China, such as the Cheung Kong Graduate School of Business), there is a possi¬ 
bility that a standardized graduate admissions test in English will be required by more Asian institutions. If that becomes 
the case, findings on what GRE scores can predict at a top-ranking Singaporean institution will offer valuable insights to 
other institutions as they consider which standardized test to use. 

Next Steps of Research 

One of the next steps of this research is to replicate the study with other institutions in Asia that receive a relatively 
large number of GRE scores from applicants. For example, institutions in Hong Kong may be suitable for the next study. 
According to the program data, more than 1,000 GRE scores were sent to two institutions in Hong Kong from 2010 to 
2011 . 

Another research direction would be to investigate an expanded list of criterion variables that GRE scores may be able 
to predict. In this study, data were not available on students’ retention in graduate programs, graduation, or performance 
beyond graduation (e.g., job placement) because the GRE revised General Test was launched in 2011 and not enough time 
has elapsed to collect such data. It is important to collect a wider range of performance outcome measures in graduate 
schools to present a complete picture of how well GRE scores predict graduate performance. 

Notes 

1 For the rest of this paper, “GRE” will be used to refer to the GRE revised General Test; for research completed prior to 2015, 
“GRE” refers to the prior version of the test. 

2 Although linear transformation to any scale would have sufficed, we converted to a 4.0 GPA scale, because it is consistent with the 
general US standard that has been used in previous validation of the GRE test. 
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Appendix A. Online Survey (Long Version of Survey) 

Educational Testing Service (ETS), the world’s largest nonprofit educational research organization, is surveying grad¬ 
uate school faculty and administrators at the NUS who might have used the GRE test to make admissions and other 
decisions. In general, these individuals are chairs of and members of admissions committees and those advised by 
admissions committees. We seek to better understand if and how the GRE test is used to make decisions at your 
institution. 

Your institution has identified you as someone who in October 2013 will participate in a focus group or interview as 
part of this study about use (or nonuse) of the GRE test. The survey will help us create more relevant questions for this 
focus group or interview. 

For your combined participation in this survey and subsequent focus group or interview, you will receive a gift card in 
the amount of $50 (50 US dollars). 

If you are interested in participating in this survey, you would visit a secure ETS website to answer online survey 
questions about your experience (or lack thereof) with the GRE test. The survey should take no more than 15 minutes 
to complete. ETS maintains the online survey. All of your answers will remain confidential and will be kept in a secure 
location at ETS. ETS will not disclose to any outside party any information from this study which would identify you 
individually. ETS will delete your identity from the data after focus groups and interviews are completed. 

You may discontinue the survey at any time, without penalty or any effect on ETS’s relationship with you. 

We hope that you will be able to assist us with this important survey. If interested, or should you have any questions or 
need additional information about the study, please e-mail Dr. David M. Klieger (one of the primary ETS investigators) 
at dklieger@ets.org. 
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We look forward to hearing from you. 

If you have read and understand this consent form, and agree to participate in the survey, please select “I AGREE” to go 
to the survey. I acknowledge that I received a copy of this form in e-mail. 

□ I Agree 

□ I Do Not Agree 

As you answer the survey questions, please consider the following: 

GRE General Test refers to the assessment of verbal, quantitative, and analytical writing skills. GRE Subject Test refers to an 
assessment of a particular domain of knowledge (Biochemistry, Cell, and Molecular Biology; Biology; Chemistry; Literature 
in English; Mathematics; Physics; or Psychology). 

“GRE Test, ” “GRE scores, ” etc. (without further specification), refer to the GRE General Test unless otherwise indicated. 

Please provide the following information: 

Family Name/Surname:_ 

Given Name:_ 

Middle Name (optional):_ 

Current Department:_ 

Current Program:_ 

Have you served (or do you serve) on an admissions committee? (Choose all that apply) 

□ I have not served on an admissions committee 

□ Served as a chair of the committee 

□ Served as a committee member 

□ Other (Please specify):_ 

For how many years did you serve as a chair of the admissions committee? 

Number of years:_ 

For how many years did you serve as a member of the admissions committee? 

Number of years:_ 

For which type of graduate degree program have you been most responsible for making admissions decisions? Choose 
only one even if you have been or are responsible for more than one: 

□ Professional master’s 

□ Research master’s 

□ Professional doctorate 

□ Research doctorate 

□ Other (Please specify):_ 

PLEASE ANSWER ALL OF THE QUESTIONS THAT FOLLOW SPECIFICALLY FOR THE GRADUATE PROGRAM 
THAT YOU CHOSE IN THE PREVIOUS QUESTION. 

Which statement best describes your admissions policy? 

□ Essentially “open door” (i.e., nonselective) 

□ Somewhat competitive (i.e., moderate credentials acceptable; may not always consider undergraduate grades or test 
scores) 

□ Moderately competitive (i.e., strong credentials required) 

□ Very competitive (i.e., some strong candidates rejected) 

□ Extremely competitive (i.e., only exceptional candidates accepted) 
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Rate the criteria below in terms of their importance for admissions decisions. 

Please note: You must provide an importance rating for all of the four criteria listed below in order to continue with the 
survey. 

Importance 



Not very 

Moderately 

Very 

Extremely 

Not used 

important 

important 

important 

important 

GRE Verbal Reasoning score □ 

□ 



□ 


□ 

□ 

GRE Quantitative Reasoning score □ 

□ 



□ 


n 

□ 

GRE Analytical Writing score □ 

□ 



□ 


□ 

n 

GRE Subject Test (e.g., Chemistry, Psychology etc.) □ 

□ 



□ 


□ 

□ 

Score most appropriate to your academic unit 








Comments(if any): 

GRE Verbal Reasoning score 








GRE Quantitative Reasoning score 








GRE Analytical Writing score 








GRE Subject Test (e.g., Chemistry, Psychology, etc.) Score most appropriate to your academic unit 




Rate the criteria below in terms of their importance for admissions decisions. 






Importance 









Not 

Not very 

Moderately 

Very 

Extremely 


used 

important 

important 

important 

important 

Quality of undergraduate institution 

n 

□ 



□ 

□ 

□ 

Undergraduate grade point average in junior and senior years or 

□ 

□ 



n 

□ 

□ 

equivalent years (e.g., grades from the last 2 years of a 4-year 








undergraduate education) 








Undergraduate grade point average in major field 

□ 

□ 



□ 

□ 

□ 

Undergraduate grade point average overall 

□ 

□ 



□ 

□ 

□ 

Undergraduate major related to field of graduate study 

□ 

□ 



□ 

□ 

□ 

Recommendation letters from faculty not known by the admissions 

□ 

n 



□ 

□ 

□ 

committee 








Recommendation letters from faculty known by the admissions 

□ 

□ 



□ 

□ 

□ 

committee 








Nonfaculty recommendations 

n 

□ 



□ 

□ 

□ 

Scores from an assessment of noncognitive skills (e.g., ETS® Personal 

□ 

n 



□ 

□ 

□ 

Potential Index (PPI)) 








Test of English proficiency for international applicants 

□ 

□ 



□ 

□ 

n 

Familiarity (i.e., applicant known by your faculty) 

□ 

□ 



□ 

□ 

□ 

Other academic achievements (presentations, publications, and projects) 

n 

n 



□ 

□ 

□ 

Personal interview 

□ 

n 



□ 

□ 

□ 

Personal statement (i.e., essay(s) written as part of the application to 

□ 

□ 



□ 

□ 

□ 

graduate or professional school) 








Prior work experience (while attending school or after graduation) 

□ 

n 



□ 

□ 

n 

Having spent time gaining life experience or practical experience after 

□ 

n 



□ 

□ 

□ 

graduating from college 








Other criteria (Please list/describe in Comments) 

□ 

□ 



□ 

□ 

□ 
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Comments (if any): 


Quality of undergraduate institution 

Undergraduate grade point average in junior and senior years or equivalent years (e.g., grades from 
the last 2 years of a 4-year undergraduate education) 

Undergraduate grade point average in major field 
Undergraduate grade point average overall 
Undergraduate major related to field of graduate study 

Recommendation letters from faculty not known by the admissions committee 
Recommendation letters from faculty known by the admissions committee 
Nonfaculty recommendations 

Scores from an assessment of noncognitive skills (e.g., ETS® Personal Potential Index (PPI)). 

Test of English proficiency for international applicants. 

Familiarity (i.e., applicant known by your faculty) 

Other academic achievements (presentations, publications, and projects) 

Personal interview 

Personal statement (i.e., essay(s) written as part of the application to graduate or professional school) 
Prior work experience (while attending school or after graduation) 

Having spent time gaining life experience or practical experience after graduating from college 
Other criteria (Please list/describe in Comments) 


Check the box beside any group below if the relative importance of the criteria listed above is different for that group. 

□ International applicants 

□ Members of underrepresented racial/ethnic groups 

□ Older applicants 

□ Applicants with a postbaccalaureate degree 

□ Female applicants 

□ Applicants with disabilities 

□ Other group (Please specify):_ 

If you selected any of the options listed above, please explain how the importance of the criteria 
differs:_ 

Do you have a formal policy regarding admissions for students with disabilities? 

□ Yes 

□ No 

If yes, please explain and/or copy and paste the policy statement:_ 

What is your policy regarding GRE General Test (Verbal Reasoning, Quantitative Reasoning, and Analytical Writing) 
scores? 

□ Required for all applicants 

□ Required but can be waived in some circumstances. 

□ Recommended for all applicants 

□ Required or recommended for some degree applicants 

□ Not required or recommended but will consider if submitted 

Please feel free to specify the circumstances and applicants for which you require or recommend GRE 

General Test scores:_ 

How important is the role of the GRE General Test in achieving the following goals? 
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Importance 



Not 

Not very 

Moderately 

Very 

Extremely 


applicable 

important 

important 

important 

important 

Matching applicants’ research interests with faculty expertise 

□ 

□ 

□ 

□ 

□ 

Selecting applicants who will be able to handle graduate school 

□ 

n 

□ 

□ 

□ 

coursework 






Selecting students who will learn important skills outside of the 

□ 

□ 

□ 

□ 

□ 

classroom (e.g., ability to run special software or lab 






equipment that facilitates research) 






Selecting applicants who will be good teaching assistants or 

□ 

□ 

□ 

□ 

□ 

instructors 






Selecting applicants whose personality and interests appear to 

□ 

n 

□ 

□ 

□ 

be a “good fit” with the culture of the field 






Selecting applicants who will promote a cohesive environment 

□ 

□ 

□ 

□ 

□ 

in the program/department/laboratory 






Maximizing gender diversity 

□ 

n 

□ 

□ 

□ 

Maximizing racial/ethnic diversity 

□ 

□ 

□ 

n 

□ 

Other goal(s) (please describe in comments): 

□ 

□ 

□ 

□ 

□ 

Comments(if any): 

Matching applicants’ research interests with faculty expertise 






Selecting applicants who will be able to handle graduate school coursework 





Selecting students who will learn important skills outside of the classroom (e.g., 





ability to run special software or lab equipment that facilitates research) 





Selecting applicants who will be good teaching assistants or instructors 





Selecting applicants whose personality and interests appear to be a 

“good fit” with 




the culture of the field 






Selecting applicants who will promote a cohesive environment in the 





program/department/laboratory 






Maximizing gender diversity 






Maximizing racial/ethnic diversity 






Other goal(s) (please describe in comments): 







How do you use GRE scores in admissions? (Choose all that apply) 

□ Applicants scoring below a specific score are not considered. 

□ GRE scores are used to categorize applicants (e.g., “probable,” “possible,” “unlikely”) before other credentials of an 
applicant are reviewed. 

□ When the applicant’s other credentials are strong, test scores are unimportant. For applicants with weaker credentials, 
the test scores are expected to compensate. 

□ Points are assigned to each applicant’s test scores and other credentials based on how important they are believed to 
be. The points are summed, and the applicants with the highest sums are offered admission. 

□ Prediction formulas (e.g., regression equations), based on test scores and other credentials of previous applicant 
groups, are used for selecting among new applicants for admission. 

□ GRE scores are used as one factor in a holistic review of applicants’ files. For example, an admissions committee might 
look at undergraduate GPA, GRE scores, and personal statements and make a decision whether to admit a candidate. 

□ Other (Please specify):_ 
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Do you use GRE scores for (select all that apply): 



GRE General Scores 

GRE Subject Scores 

Academic advising? 

□ 

□ 

Awarding assistantships and fellowships? 

□ 

n 

Placement of students in courses? 

□ 

□ 

Comprehensive examination or other graduation requirement? 

n 

□ 

Another purpose? (Please specify): 

□ 

n 


If you selected another purpose, please specify:_ 

How do you use multiple sets of GRE scores that an applicant sends? 

□ Use the most recent scores 

□ Use the highest score from each GRE section across test administrations (i.e., “mix-and-match” highest scores); 

□ Use the lowest score from each GRE section across test administrations (i.e., “mix-and-match” lowest scores); 

□ Average the scores from each GRE section across the test administrations 

□ Other (Please specify):_ 

How important is each of the listed reasons for your not requiring or recommending GRE test scores for admissions? 
(These are based on a belief or perspective that an institution, department, program, committee, or individual could 
have.) 



Unimportant 

Moderately 

important 

Very 

important 

Test scores do not predict performance well 

□ 

□ 

□ 

The test poses special difficulties for those with disabilities 

□ 

□ 

n 

Test scores do not predict equally well across demographic groups 

□ 

□ 

n 

Test scores are redundant with other information we receive 

□ 

□ 

n 

Other information received from the applicant is more important 

□ 

□ 

□ 

Almost all applicants are admitted 

□ 

□ 

n 

The GRE tests appear to penalize the better, more creative students in our academic unit 

□ 

□ 

n 

GRE Subject Test content is not appropriate for our use 

□ 

□ 

n 

Requiring or recommending GRE scores would add more work to an already complicated 

□ 

□ 

n 

process 

Requiring or recommending GRE scores would add more cost to the application process 

□ 

□ 

□ 

Other reason. Please specify in comments 

□ 

□ 

□ 


Comments(if any): 


Test scores do not predict performance well 

The test poses special difficulties for those with disabilities 

Test scores do not predict equally well across demographic groups 

Test scores are redundant with other information we receive 

Other information received from the applicant is more important 

Almost all applicants are admitted 

The GRE tests appear to penalize the better, more creative students in our academic unit 
GRE Subject Test content is not appropriate for our use 

Requiring or recommending GRE scores would add more work to an already complicated process 
Requiring or recommending GRE scores would add more cost to the application process 
Other reason. Please specify in comments 


Background Questions (Optional) 

Gender: 

□ Female 

□ Male 

What is your age (in years)?_ 
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Please provide your country or countries of citizenship. You can indicate up to two countries from the drop-down menus 
below. 

□ Abkhazia—Republic of Abkhazia ... 

□ Zimbabwe—Republic of Zimbabwe 

Do you possess significant work experience outside of academia such as in industry, government, or the nonprofit 
sector? 

□ Yes 

□ No 


Appendix B. Interviews/Focus Groups Script 

National University of Singapore Focus Group 

October 2013 

Introduction 

[As participants are joining the group, ask them to fill out the short-form survey (name, department, and some information 
about if and how the participant’s program uses the GRE test)]. 

Hi, my name is David, and I’m from Educational Testing Service (ETS), the largest nonprofit educational research company 
in the world. As you might know, ETS’s main location is in the United States, specifically Princeton, New Jersey. Today, I’m 
here today to learn about if and how the GRE tests are used and thought about at your university. This is part of a larger 
effort to better understand how the GRE test is used and perceived outside of the United States. I’m especially interested 
in hearing about it from your perspectives, because as participants you’re the experts. 

During the next hour or so. I’ll be asking you a series of questions about your experiences regarding the GRE tests. 
Those tests consist of the GRE General Test and Subject Test. The GRE General Test assesses Verbal skills and Quantitative 
skills through multiple choice and analytical writing skills through constructed responses. The GRE Subject Tests measure 
domains of knowledge such as Biology and Psychology. 

There are no right or wrong answers to what I ask about — I’m just really interested in your ideas, impressions, and 
opinions about the GRE. 

As we talk today, there are a few things you should know: 

• First, I’d like to tape record the discussion because your ideas are important to me and I really want to remember the 
things you say. However, this is a private conversation, so the only people listening to the tape will be a few people 
I work with at ETS. If you say something especially interesting (which you all will) and I want to quote what you’ve 
said in something I write, I’ll give you a fake name, so it’s anonymous. The information you provide will be used for 
research purposes only. 

• If I ask a question that you don’t want to answer, that’s okay—you don’t have to. 

• Also, to be sure that the group is private. I’m going to ask that you not share specific things that have been said in 
the group with GRE staff or friends in the program who are not here right now. You can tell people about any of the 
questions that I ask but try not to share personal things said by other people in the group. Respecting everyone’s 
privacy and different opinions is probably the most important thing about focus groups. 

• And last, since I’m using a tape recorder, please try to speak one at a time, because I won’t be able to hear what’s said 
on the tape if people speak over each other. So if you can just try to speak clearly and wait until others are finished 
talking before sharing your thoughts that would be great. I really want to learn about each of your opinions. 

Do you have any questions for me before we begin? 

Questions for Programs That Use the GRE Test (With Follow-Up or Question Skipping, as Appropriate) 

1. In your program’s decision-making processes, how important are the GRE tests, and why? I am interested in your 
answer in a relative sense (e.g., GRE relative to undergraduate grades, letters of recommendation, and undergraduate 
institution). 
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2. If you are in a program that is highly quantitative, how much do you care about the Verbal section or the analytical 
writing section of the GRE General Test? Conversely, if you are in a relatively nonquantitative program, how much 
do you care about the quantitative section of the GRE General Test? 

3. Do you think that your communication to potential applicants that you use the GRE test influences their decision 
about whether or not to apply to your program? If so, how? 

4. What do you think that the GRE General Test and Subject tests best help you to do (e.g., select students who will be 
able to handle graduate school coursework)? 

5. Do you think that the GRE General Test and Subject tests play a different role for Singaporean graduate programs 
than they do for US ones? If so, how? 

6. What are the important skills and other characteristics for success as a graduate or professional student in your pro¬ 
gram? In what ways could your students be better prepared for graduate or professional school? 

7. Do you think the GRE tests could be improved? (Yes or No). If “yes,” then why and how could they be improved? 

8. What do you do if you receive multiple sets of GRE scores for an applicant? Do you average them, take the highest 
scores, use the most recent scores, or do something else? Why? 

9. Why does your program use GRE scores the way that it does (e.g., in a compensatory way if an application is not 
strong vs. with a firm cut score) ? 

Questions for Programs That Do Not Use the GRE Tests (With Follow-Up or Question Skipping, as Appropriate) 

1. Why don’t you use the GRE Tests? Let’s talk about the GRE General Test and then the Subject tests. 

2. Could the GRE Tests be changed in such a way that you would use them? How? 

3. What are the important skills and other characteristics for success as a graduate or professional student in your pro¬ 
gram? 

4. In what ways could your students be better prepared for graduate or professional school? 

5. If you used the GRE test, would that affect who applies to your program? If so, how? 

Thank you for your participation! This information will help ETS better understand how you view and use the GRE 
tests. 

If you have any questions or would like to talk to me further, please don’t hesitate to do so now or at a future time. I 
am happy to give you my contact information. I’ll be in Singapore during this week. 


Appendix C. Univariate Range Restriction Adjustment 

The univariate adjustment for range restriction is based on the following formula: 

U X P2 

u 2 x -i)p 2 2 + i 

where 

p x = validity coefficient adjusted for range variation 
p 2 = observed (unadjusted) validity coefficient 

^x 'U1 1^x2 

where rr xl is the standard deviation for the applicant pool and a x2 is the standard deviation for enrolled students (Hunter 
& Schmidt, 2004, p. 107). 
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